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Scientists have made great strides in modeling physical processes that repre- 
sent various weather and climate phenomena. Many modeling systems that 
represent the major earth system components (the atmosphere, land sur- 
face, and ocean) have been developed over the years. However, developing 
advanced Earth system applications that integrates these independently de- 
veloped modeling systems have remained a daunting task due to limitations in 
computer hardware and software. Recently, efforts such as the Earth System 
Modeling Framework (ESMF) and Assistance for Land Modeling Activities 
(ALMA) have focused on developing standards, guidelines, and computa- 
tional support for coupling earth system model components. In this article, 
the development of a coupled land-atmosphere hydrometeorological model- 
ing system that adopts these community interoperability standards, is de- 
scribed. The land component is represented by the Land Information System 
(LIS), developed by scientists at the NASA Goddard Space Flight Center. 
The Weather Research and Forecasting (WRF) model, a mesoscale numerical 
weather prediction system, is used as the atmospheric component. 

LIS includes several community land surface models that can be executed 
at spatial scales as fine as 1km. The data management capabilities in LIS 
enable the direct use of high resolution satellite and observation data for 
modeling. Similarly, WRF includes several parameterizations and schemes 
for modeling radiation, microphysics, PBL and other processes. Thus the 
integrated LIS- WRF system facilitates several multi-model studies of land- 
atmosphere coupling that can be used to advance earth system studies. 

Significant Findings 

In this article, several synthetic and real simulations using the coupled LIS- 
WRF system are presented. The results suggest that capturing the fine scale 
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heterogeneities associated with topography, soils, land cover, and other land 
surface features is important for accurate characterization of the land surface 
boundary. Further, the accurate representation of land surface conditions 
have significant impacts on the evolution of clouds and precipitation. The 
wealth of modeling infrastructure and tools in the LIS-WRF system is ex- 
pected to enable studies to investigate the nature of interaction and feedback 
between land and the atmosphere. 
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Abstract 

Interactions between the atmosphere and the land surface have considerable influ- 
ences on weather and climate. Coupled land-atmosphere systems that can realis- 
tically represent these interactions are thus critical for improving our understand- 
ing of the atmosphere-biosphere exchanges of energy, water, and their associated 
feedbacks. NASA’s Land Information System (LIS) is a high resolution land data 
assimilation system that integrates advanced land surface models, high resolution 
satellite and observational data, data assimilation techniques, and high performance 
computing tools. LIS has been coupled to the Weather Research and Forecasting 
(WRF) model, enabling a high-resolution land-atmosphere modeling system. These 
coupled Earth system model components help us to understand and predict regional 
and global water and energy cycles. Synthetic and realistic simulations using the 
coupled LIS- WRF system for a June 2002 International H20 Project (ISOP) case 
study illustrate the impact of accurate, high resolution land surface conditions on 
the evolution of clouds and precipitation. 
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1 Introduction 


Accurate characterization of the complex dynamics governing land- atmosphere interactions 
is important for improving, our understanding of water and energy cycles. The land surface 
exerts significant influences on the atmospheric processes, across a number of spatial and 
temporal scales through the exchanges of mass and energy (Pan and Mahrt (1987); McCorcle 
(1988); Castelli et al. (1996)). The dependence and sensitivity of the atmospheric processes 
to the land surface boundary has been reported in many studies (Pielke (2001); Avissar 
and Pielke (1991); Chen and Avissar (1994)). For example, soil moisture, a variable that 
represents the amount of available water on the land surface, modulates the partitioning the 
surface energy into turbulent components of sensible and latent heat fluxes. The evolution 
of atmospheric processes such as precipitation, which supplies water to the land surface, is 
in turn is intimately linked to the moisture state of the surface, This interdependence has 
been cited as the cause of anomalous features such as drought and flood (Oglesby (1991); 
Koster et al. (2000)). Several past studies have shown the effects of surface heterogeneity 
and soil moisture gradients on the development of convection (Pielke (2001); Shaw et al. 
(1997); Holt et al. (2006); Trier et al. (2004)). Holt et al. (2006) further demonstrated the 
influence of the effects of vegetation and soil processes on convection and suggest that a 
detailed representation of these processes should be included in forecast models. 


Though the interdependence of land surface conditions and atmospheric processes have been 
well-established (Betts and Ball (1995); Basara and Crawford (2002)), the initialization of 
mesoscale and NWP models have been limited by the lack of availability of real-time land 
surface states such as soil moisture, temperature, and snow pack. Traditional ground-based 
observation systems provide “point” measurements of soil moisture, which are inadequate 
for application and interpretation at varying spatial scales. Land surface initialization in re- 
gional mesoscale models is typically performed with coarser, spatially interpolated analysis 
from operational streams. The coarse representation of surface conditions fails to capture 
the fine scale heterogeneities associated with topography, soils, land cover and other land 
surface features. The interpretation of soil moisture and temperature profiles to the land 
surface model’s configuration further introduces errors. The mesoscale model simulation is 
also constrained to use the same land surface model and the set of parameters to maintain 
consistency between initialization and forecasts. Uncoupled or offline land surface modeling 
systems have been shown to adequately capture the evolution of land surface states (Robock 
et al. (2003)), which in turn can be used to initialize the atmospheric models. Offline systems 
to generate high resolution land surface initial conditions such as High Resolution Land Data 
Assimilation System (HRLDAS; Chen et al. (2004)) have also been used to demonstrate the 
impact of fine scale soil moisture representation on forecasts of deep convection. This sys- 
tem, however, currently supports a single land surface model, a limited set of land surface, 

. observational and meteorological data; and certain limited domains. NASA’s Goddard Space 
Flight Center (GSFC) has recently developed a Land Information System (LIS; Peters-Lidard 
et al. (2004); Kumar et al. (2006)) capable of simulating land surface conditions at various 
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spatial resolutions as high as 1km globally and as fine as 30m regionally. LIS is a Land 
Data Assimilation System (LDAS) that consists of software interfaces to execute several 
community land surface models using a blend of observationally-based precipitation, radia- 
tion, and other model-based sources of meteorological inputs and surface parameters. The 
LIS infrastructure generates spatially and temporally distributed estimates of land surface 
conditions using previously observed or model-derived meteorology to constrain and force 
the land surface models in an uncoupled manner. 

LIS is a modeling system that provides both the ability to resolve and evaluate the impact 
of the heterogeneities in vegetation, soils, topography and other land surface states at the 
specified resolution, and the ability to specify custom land surface models and parameters. In 
this article, we describe the extension of the uncoupled LIS framework to support a coupled 
land-atmosphere modeling, which can employ several land surface models and can be applied 
over the region of interest. The flexibility provided by LIS can advance earth system science 
by facilitating multi-model studies of land-atmosphere coupling. The explicit characterization 
of the land surface at the same spatial scales as that of cloud and precipitation processes 
helps in accurately characterizing the land-atmosphere interaction. 

An integrated system to conduct high resolution land-atmosphere simulations is enabled by 
coupling LIS and the Weather Research and Forecasting model (WRF; Michalakes et al. 
(2001)). WRF is a state of the art mesoscale numerical weather prediction system that 
serves both the atmospheric research and operational forecasting communities. LIS is used 
in an uncoupled manner to generate high resolution land surface initial conditions for a 
coupled simulation. Using LIS as the land surface modeling component in WRF in coupled 
simulations also allows for consistency across models, parameters, and computational con- 
figurations. In the sections that follow, we describe LIS, WRF and the architecture of the 
coupled system. A number of synthetic and real case day simulations demonstrating various 
modeling capabilities enabled by the coupled system are presented in section 5. Finally a 
summary and future extensions to the system are presented. 


2 Land Information System 


Land surface processes constitute significant components of the terrestrial climate system 
and they have profound influences on the interactions between the biosphere and atmo- 
sphere. Land surface models provide characterizations of the water and energy exchanges 
and biogeochemical processes of the soil-vegetation-snowpack medium. A realistic represen- 
tation of these processes is critical for improving the understanding of the boundary layer 
and land- atmosphere interactions. The development of LIS has been motivated by the need 
to develop an infrastructure that combines the use of land surface simulation, available 
observations and the required computing tools for accurate land surface prediction. As dis- 
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cussed in Kumar et al. (2006), LIS integrates and extends the capabilities of Land Data 
Assimilation Systems (LDASs) such as the 25km Global Land Data Assimilation System 
(GLDAS; Rodell et al. (2004)) and the 12.5km North American Land Data Assimilation 
System (NLDAS; Mitchell et al. (2004)). LIS is primarily an infrastructure for operating an 
ensemble of land surface models with capabilities for data integration and assimilation, over 
user-specified regional or global domains. The new phase in LIS development is to extend its 
capabilities by linking with other earth system components, enabling coupled systems that 
can model land-atmosphere interactions more effectively. 

LIS is designed using advanced software engineering principles, and features a highly mod- 
ular, flexible, object oriented, component-based framework. Figure 1 shows the software 
architecture of LIS. The core of the system consists of structures to manage generic utilities 
such as time, configuration, geospatial transformations, I/O, parallel computing constructs, 
logging, etc. These structures provide generic, model-independent support for high perfor- 
mance computing, resource management, data and I/O handling, and other functions. The 
LIS core controls the overall program execution and manages the inclusion of user-defined 
extensible components through several related abstractions. These abstractions, shown in 
the middle layer, include generic representations of land surface models, data assimilation 
schemes, meteorological forcing schemes, domains, running modes etc. The specific user de- 
fined components extend these abstractions. For example, Figure 1 shows a number of land 
surface models (Noah, CLM, HySSIB, Catchment) implemented in LIS through the land 
surface model abstraction. By providing a structure that allows the reuse and community 
sharing of modeling tools, LIS allows rapid prototyping and development of new applica- 
tions. These interoperable features in LIS has enabled the incorporation of a growing suite 
of community LSMs, meteorological forcing analyses, different sources of land surface pa- 
rameters, and data assimilation schemes. The system also allows for the plug and play of 
various user-defined components and has enabled several intercomparison studies involving 
land surfacejnodels, parameters, and assimilation schemes (Dirmeyer et al. (1999)). 


3 Weather Research and Forecasting Model 


The Weather Research and Forecasting (WRF; Michalakes et al. (1998, 2001)) model is a 
next-.generation mesoscale numerical weather prediction system that serves both operational 
and research communities. The system consists of multiple dynamical cores, preprocessors for 
producing initial and lateral boundary conditions for simulations, and a three-dimensional 
variational data assimilation (3D VAR) system. WRF is built using software tools to enable 
extensibility and efficient computational parallelism. The use of WRF system has been re- 
ported in a variety of areas including storm prediction and research, air-quality modeling, 
wildfire, hurricane, tropical storm prediction, and regional climate and weather prediction 
(Michalakes et al. (2004)). 
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The WRF software follows a modular structure with complex functionalities encapsulated , 
into three main hierarchical layers. The top level corresponds to the driver layer and the low- 
est level corresponds to the model layer. The mediation layer provides the interface between 
the driver and the model layers. The driver performs the overall creation and organization 
of model domain data structures, control of model clock, parallel decomposition functions, 
processor topologies, and other aspects of parallelism. The management of I/O operations 
on model domains, control of forcing, feedback and nested operations are also handled in 
the driver layer. The mediation layer represents the instances of a particular dynamical core 
and its interaction with the model layer. This layer manages the invocation of interprocess 
communication and multithreading. The model layer encompasses the actual computational 
modules that constitute the model. These include advection, diffusion, physical parameteri- 
zations and representations of various other processes. 


4 Design of the Integrated Coupled Land Atmosphere System 


The advanced features of LIS and WRF are combined by integrating these modeling systems 
into a single system for coupled hydrometeorological modeling. In the coupled system, LIS 
and WRF represent the components of the land surface and the atmosphere, respectively. 
The design of the coupled system is enabled by the structure and tools provided by the Earth 
System Modeling Framework (ESMF; Hill et al. (2004)). ESMF is a project intended to de- 
velop standards-based, open-source software tools to enable software reuse, interoperability 
and performance portability in Earth Science Applications. The ESMF software primarily 
consists of a superstructure for coupling and exchanging data between components (e.g., 
atmosphere, land) and an infrastructure consisting of tools and utilities to speed up con- 
struction of components and to ensure consistent, guaranteed component behavior. A model 
component using ESMF is organized to perform three major functions: initialization, run, 
and finalization. The initialization methods typically implement the initialization of param- 
eters, initial and boundary conditions, and any other model setup. The run method provides 
the model simulation methods and the finalization routine provides methods to properly 
complete the model operations. 

As shown in Figure 1, the LIS software architecture allows LIS to be used in an analysis or 
uncoupled mode, where the meteorological boundary conditions are provided using existing 
meteorological data. In the coupled or forecast mode of operation, LIS obtains the mete- 
orological boundary conditions from the atmospheric component such as WRF. Figure 2 
illustrates the roles of LIS in the coupled LIS- WRF system: LIS is used as in an analy- 
sis mode to provide initial land surface conditions for WRF, and in the coupled mode for 
forecast/coupled simulations, LIS acts as the land surface component. 

In the coupled mode of operation, the sequence of data exchanges between WRF and LIS 


5 



axe shown in Figure 3. In the LIS-WRP system, LIS and WRF are represented as two model 
components lisComp and wrfComp, respectively. lis2wrfCpl and wrf 21isCpl represent the 
coupler components that perform the actual data transformations and exchanges between the 
model components. The user-defined components that conform to ESMF use special objects 
called ESMF_State for inter component data exchanges. Every component accepts one or more 
ESMF_States as import states and produces one or more ESMF -States as export states. The 
data exchanged between the model components are encapsulated by lislmport, lisExport, 
wrf Import, and wrf Export objects that represent the import and export states from LIS and 
WRF. The simulation starts at t=tO and cycles n times. At the beginning of the simulation, 
both model components and the coupler components are created. The invocation to LIS is 
performed from the WRF surface driver, which is a model layer component in WRF. The 
export states from WRF are fed into LIS as the import states and the lisComp is executed. 
Finally the export states from LIS are fed back to WRF and the process repeats at every 
invocation of the surface driver. 

The import and export states for the LIS model component follows the Assistance for Land 
Modeling Activities (ALMA (2002)) convention. ALMA is a data convention to facilitate the 
exchange of forcing data for land surface models and the results produced by them. In the 
design described above, the ALMA convention is imposed on the import and export states 
of LIS. The lislmport state consists of the atmospheric forcing variables that are specified 
in the ALMA data convention for forcing data. Similarly, the lisExport state is defined 
in accordance with the ALMA standard model output convention. In order for any LSM in 
LIS to be used in a coupled simulations, the LSM needs to provide methods to define the 
lisExport state, and methods to translate the lislmport state to its own variables. Thus, 
the two interfaces in LIS that define the import and export states of a LSM are sufficient to 
complete its incorporation in the coupled system. 

The integration of LIS and WRF provides an enhanced system with several improved func- 
tionalities. The LIS- WRF system is designed such that LIS acts as the land modeling com- 
ponent, encapsulating several community land surface models, the broad set of data and 
land surface data assimilation tools. The data components in LIS enable the direct use of 
high resolution satellite and observational data streams for modeling. Similarly, WRF acts 
as the atmospheric component, providing a number of surface layer, Planetary Boundary 
Layer (PBL) schemes and parameterizations. The interaction of these two modeling systems 
through generic interfaces allows the plug and play of different land models, PBL and surface 
layer schemes. Further, the conformance to the ALMA standard and ESMF structure allows 
the model components of LIS and WRF to interface with other compatible earth system 
models. 

Land surface states simulated by the LSMs typically require long-term simulations to reach 
thermodynamic equihbrium with the meteorology. This modification process or “spinup” 
of the model that adjusts for initial anomalies in soil moisture content or meteorology is 
important for accurate characterization of land surface conditions (Yang et a! (1995)). Fur- 
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ther, Rodell et al. (2005) notes that the model climatologies differ from those observed in 
nature, and an ideal initialization of a LSM should be done with the results from a long-term 
simulation. Studies have shown that model biases originating from improper spinup can ad- 
versely impact the laud surface simulations (Maurer and Lettenmaier (2004); Koster and 
Suarez (2001); Zhang and Erederiksen (2003)). LIS can be executed in an uncoupled mode, 
forcing the LSM with a meteorological dataset to conduct multiyear spinup simulations. The 
configurable features in LIS enable these simulations to be conducted at the same grid con- 
figurations using the same parameters and LSM as those used in the coupled simulations. 
Further, the high performance computing infrastructure in LIS allows these simulations to 
be conducted rapidly, with a turnaround time on the order of hours for a multi-year spinup 
(depending on the processor speed and availability). 

WRF supports a horizontal nesting option that allows a high resolution simulation to be 
focused over a region of interest by introducing additional grids that are used to provide 
lateral boundary conditions to the inner, finer grid.. In order to preserve consistency across 
different processes, the surface fluxes and processes also need to be represented at the same 
spatial scales. LIS provides options to set up regional domains in the same grid configurations 
as that generated by the WRF preprocessor (Standard Initialization - SI) program. LIS allows 
these domains to be set up anywhere in the world, and conduct longterm simulations using 
suitable meteorological forcing and available parameters. 


5 Results 


In this section, we present the results of simulations and experiments conducted using the 
coupled LIS- WRF system. These simulations illustrate the improvements in modeling capa- 
bilities enabled by the enhancements in computing and software of the coupled system. This 
section is organized into three segments: In the first segment, a set of idealized synthetic ex- 
periments to to test the land-atmosphere coupling interface is presented. Section 5.2 presents 
the application of the system to forecast a case day. Finally an analysis of the computational 
performance and scalability of the system is presented. 


5.1 Synthetic Experiments 


The goal of the synthetic experiments is to examine the exchange of energy and water across 
the land-atmosphere interface using a number of simple idealized cases. The experiments 
are designed using horizontally homogeneous conditions and a steady atmospheric forcing. 
The modeling domain is chosen over a region in the U.S. Southern Great Plains, which was 
also the focus of large weather field experiment called the International H20 project (IHOP- 
2002; Weckwerth et al. (2004)), conducted from 13 May to 25 June, 2002. The domain is 
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chosen to be of size 100x100 grid points at 1km resolution, with 41 vertical levels. The height 
of the vertical levels is chosen with high resolution near the ground and progressively less 
resolution with increasing height. The whole domain is uniformly specified with grassland 
as the landcover type. The soil is uniformly specified with sandy clay loam, which is the 
dominant soil type of the area. The experiments are conducted with two of the land models 
in LIS; the community Noah LSM (Noah; Ek et al. (2003)), and the Community Land 
Model Version 2.0 (CLM; Dai et al. (2003)). Both these models dynamically predict water 
and energy fluxes and states at the land surface, although their parameterizations and/or 
structures representing various processes differ. For example, CLM includes 10 layers for soil 
temperature and soil water, with explicit treatment of liquid water and ice, whereas Noah 
uses four soil layers. An important difference between the two is related to the representation 
of vegetation and its control over transpiration: Noah assumes uniform Leaf Area Index 
(LAI) over the globe, with a temporally and spatially varying Green Vegetation Fraction 
(GVF) and a simple Jarvis-based stomatal resistance formulation controlling the maximum 
transpiration from vegetated grids or tiles. In contrast, CLM has a much more complex 
representation of the canopy, including variable LAI and coupled photosynthesis-stomatal 
conductance formulation. 


Two sets of experiments are conducted: A DRY experiment where the coupled system is 
initialized with a dry, soil moisture deficient land surface, and a WET experiment with a 
relatively wet soil state. Atmospheric initial conditions for the modeling domain are uniformly 
applied from the sonde extracted from the data archive of IHOP-2002, for June 6, 2002 at 
12GMT. This day is chosen since there were light winds in the PBL and the atmosphere 
was extremely dry leading to negligible moist development. These conditions are generally 
favorable to the growth of mesoscale circulations that can result from the influence of the 
land surface conditions as noted by Lynn et al. (2001). 


Figures 4 and 5 show the domain averaged surface energy fluxes from the DRY and WET 
simulations, respectively. From the figures it can be observed that the LSMs generate the 
partitioning of radiant energy between the turbulent (latent Qle and sensible Qh) heat 
fluxes and the ground heat (Qg) fluxes differently. The partition of the input energy is also 
tightly coupled to water availability at the surface. In the DRY simulations, the plants are 
constrained by the lack of transpirable water, leading to stomatal closure. As a result, the 
surface latent fluxes in the DRY experiments are expected to be zero. The DRY experiments 
using both LSMs demonstrate this feature. In the WET experiments, the latent heat flux 
constitutes the primary component of the turbulent fluxes, due to the evaporation of water 
through the plant stomata. Both LSMs exhibit these trends, though the partition is done 
differently. Figure 6 shows the evolution of planetary boundary layer (PBL) height from 
the DRY and WET experiments. The PBL, the lowest portion of the atmosphere is strongly 
influenced by surface characteristics. This influence is demonstrated in the experiments using 
both LSMs. The PBL height from the DRY experiment is larger than the height from the 
WET experiment, indicating the direct influence of larger sensible fluxes on the PBL growth. 
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These simple, synthetic experiments demonstrate how the evolution of fluxes of energy and 
water between the land-atmosphere interface is intimately linked to the moisture state of 
the land surface. The atmospheric processes that supply moisture to the surface are in turn 
dependent on the moisture-driven fluxes from the land. 


5.2 Impact of LIS on precipitation forecast 


To demonstrate the impact of high resolution land surface conditions on the development 
of clouds and precipitation processes, the LIS-WRF system is applied over the IHO P-2002 
region using June 12, 2002 as the case day. This date is chosen since the day was characterized 
by clear skies and relatively weak synoptic forcing. As a result, the. land surface influence is 
likely to be magnified in the evolution of mesoscale circulations. The data archives of IHOP 
show that convection occurred between 12GMT on June 12, 2002 through 12GMT on June 
13, 2002. 

The chosen modeling domain is centered around the area where the convection is initiated. 
The model domain shown in Figure 7 consists of three nested domains, with the outermost 
grid (dl) with a 9-km grid spacing, and the inner grid (d2) with 3km and the innermost 
grid (d3) with a 1km spacing. The dl domain consists of 399x270 grid points, the d2 domain 
consists of 402x402 grid points and the d3 domain consists of 504x504 points. The vertical 
grid consists of 41 vertical levels, with the lowest level at 10m extending upto 18km above 
the ground level. The 32km North American Regional Reanalysis (NARR; Mesinger et al. 
(2006)) data is used to initialize the meteorological conditions for all three nests. Other 
WRF model settings includes Lin Microphysics, RRTM longwave, and Dudhia shortwave 
parameterizations. 

Three different simulations are conducted for the case day using initial land surface conditions 
generated by: (1) from a climatology, (2) from a long term spinup of the Noah land surface 
model in LIS, and (3) from a long term spinup of the CLM land surface model in LIS. 
The climatology-based soil moisture and temperature profiles are derived from the NARR 
data by the WRF preprocessor program, which is on a coarse, 32km grid. The soil profiles 
in LIS are generated from the uncoupled, long-term spinup by LIS. Using Noah and CLM 
LSMs, each of the three nested domains are spunup for approximately 17 years, starting 
from 1985, using bias-corrected atmospheric reanalysis data (Berg et al. (2003)). From 2000 
onwards the inner nests are forced with the forcing data used in the NLDAS (Cosgrove et al. 
(2003)), which includes the radiation from the GOES satellites and the precipitation from 
the NCEP STAGEIV estimates (Lin and Mitchell (2005)). The spatial extent of the NLDAS 1 
forcing, did not cover the dl domain completely. As a result, the dl domain is forced with the 
output from the Global Data Assimilation System (GDAS), the global operational weather 
forecast model of NCEP (Derber et al. (1991)). A static, 1km resolution global vegetation 
classification dataset produced at the University of Maryland (Hansen et al. (2000)) is used 
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to prescribe vegetation-based parameters for both LSMs. The Noah LSM requires monthly 
GVF and quarterly snow-free albedo. The GVF fields in LIS are currently sampled from the 
global, 0.144 degree (approx.' 15-km), monthly 5-year climatology derived by Gutman and 
Ignatov (1998) from the Advanced Very High Resolution Radiometer (AVIIRR) instrument 
aboard the NO A A satellites. CLM requires climatologies of Leaf Area Index (LAI) and the 
simulation uses a 1km, 4 year average climatology derived from the MODIS Collection 4 LAI 
products. Soil hydraulic properties are derived using vertically uniform sand, silt and clay 
percentages from the 5-min global soils dataset of Reynolds et al. (1999), which is also known 
as the Food and Agriculture Organization (FAO) based data. This dataset is used in the 
outer domain dl. A finer, lkm resolution State Soil Geographic Database (STATSGO) soil 
parameters from Miller and White (1998), which carries 16 texture classes over 11 layers to 
2-m depth, available for the continental United States is used for the inner domains d2 and 
d3. In the simulations using the Noah LSM, water points are treated as quasi-ocean points 
using redefined sea surface temperature as the skin temperature value. In CLM, however, the 
water points are modeled with an explicit lake model (Bonan (1996)), with lake temperatures 
computed based on a six-layer model. 

Figures 8 and 9 show a comparison of the initial soil moisture and soil temperature fields for 
the d3 domain from WRF preprocessor program (which is based on the NARR data), LIS 
Noah spinup, and the LIS CLM spinup, at 12Z, June 12, 2002, which is the initial time of 
the forecast simulation. The comparison demonstrates the fine-scale features from the LIS 
spinups, compared to the overall more uniform NARR data-based initial conditions. Further, 
it can be observed that the CLM spinup demonstrates more fine scale features, compared 
to Noah. This can be attributed to the fact that the CLM spinup uses the lkm MODIS 
LAI along with other high resolution data to prescribe the input land surface parameters, 
whereas Noah LSM uses a rather coarse GVF data to prescribe the vegetation phenology. 

The initial soil moisture and temperature profiles of 12Z June 12, 2002, used in the three simu- 
lations are evaluated by comparing them to the station data from the Oklahoma mesonet Brock 
et al. (1995). Table 1 shows the summary of the comparisons that includes approximately 
115 stations. It can be observed that the LIS generated soil moisture and temperature initial 
conditions demonstrate reduced error estimates compared to the NARR-based initialization. 
The bias estimates for soil moisture indicates that the NARR and LIS-Noah initializations 
are hotter and drier, whereas the LIS-CLM initialization is wetter and colder. 

The simulations using the above described initial conditions are used to conduct a 24 hour 
coupled LIS- WRF simulation starting at 12Z, June 12, 2002. These simulations are analyzed 
for their predictive skill in generating the case day convection. Table 2 shows a summary of 
several evaluation metrics typically applied for NWP rainfall verifications. The NARR-based 
simulation overpredicts the precipitation event, indicated by a high positive bias. The bias is 
reduced in the LIS-Noah simulation. The LIS-CLM run has a negative bias. The root mean 
square (RMS) errors are highest in the NARR simulation and are reduced in the LIS-based 
simulations, with the LIS-CLM simulation showing the lowest error estimate. Probability 
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of Detection (POD), which measures the success of the forecast in correctly predicting the 
occurrence of rain events is improved with the LIS-based forecasts, with the LIS-CLM run 
showing the best metric. Similarly, the False Alarm Ratio (FAR) , which measures the fraction 
of event forecasts that are actually non-events, is reduced in LIS-based simulations, the LIS- 
CLM run with the lowest FAR value. The simulations are also evaluated by comparing 
the distribution of rain rates with the STAGED/ rates, shown in Figure 10. The NARR 
data-based simulation underestimates the low rain rates, and overpredicts the higher rain 
rates. The LIS-based simulations provide estimates of rain rates closer to the observations, 
compared to the NARR data-based simulation. 

These results demonstrate the improved skill in forecasts with the use of LfS-derived land 
surface initial conditions. Further, the improved estimates in skill, magnitude and distribu- 
tion of the CLM forecast can be attributed to the more physically-based, process oriented 
parameterizations in its treatment of vegetation, soils, and water components of the land 
surface and the use of more high resolution land surface parameters compared to the Noah 
LSM. 


5.3 Computational Performance 


The real case simulation described above is extremely computationally demanding. The sim- 
ulation is carried out using 128 processors, requiring approximately 50 gigabytes of memory. 
The 24 hour simulation required approximately 22 hour wall clock time. .Therefore, it be- 
comes very important that the coupled system scales well in a multiprocessor environment 
in order to have a realistic turnaround time. 


To analyze the computational scalability of the LIS-WRF system and to examine the over- 
head imposed by LIS, simulations are carried out on a 100x100 domain at 1km resolution, 
varying the number of processors, using the LIS-WRF system and the native WRF system, 
using the Noah land surface model. All simulations are carried out on the HP / Compaq SC45- 
halem system at the NASA Goddard Space Flight Center. This supercomputing system em- 
ploys standard symmetric multiprocessor (SMP) nodes that incorporates four Alpha- EV68 
processors. The nodes are connected to a Single rail Quadrics QsNet network that yields 
peak internode bandwidth of about 280MB/s. Figure 11 shows the comparison of execution 
times for a 24 hour forecast for different number of processors, for both the LIS-WRF, and 
the native WRF systems. It can be observed that the use of multiprocessors provides sig- 
nificant improvement in computational performance and the execution scales efficiently in a 
massively parallel environment. Further, the overhead imposed by LIS is in the order of 5-7 
percent and can be considered to be minimal. 
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6 Summary 


The article describes the design of a high resolution hydrometeorological modeling system 
that integrates LIS and WRF using the interoperable standards and constructs provided 
by ESMF. This highly modular system is designed using advanced software engineering 
principles and practices. The wealth of user-defined extensible components in LIS and WRF 
enables prototyping, plug and play, and intercomparison of various model components. The 
use of LIS as the land surface modeling component in WRF enables not only the use of 
different land surface models, but also the use of high resolution data access and distribution, 
high performance computing, and land surface data assimilation tools. 

The land surface initial conditions for the coupled LIS- WRF simulations are generated by 
running LIS “uncoupled” for 17 years, using observationally-based precipitation, radiation, 
and meteorological inputs and high resolution surface parameters. Long uncoupled Simula- • 
tions are necessary because deep soil moisture and temperatures require long integrations 
to reach dynamic equilibrium. Our case study results suggest that the soil thermodynamic 
profiles generated by LIS improves the coupled system estimates relative to the standard 
initialization using the output from an NWP model. 

The integration of these two complex systems is achieved without a significant overhead in the 
computational scaling performance. The analysis of the coupled system simulations showed 
that the code scaled very well in a massively parallel environment. Though the real case day 
simulation described in the article is computationally intensive, near-real time performance 
is achieved using adequate computing resources. 


In this article, the interoperability of using different LSMs is demonstrated using a number of 
synthetic experiments. Many studies to investigate the impact of high resolution satellite and 
ground observations such as those provided by MODIS can be readily investigated through 
the use of LIS. The data assimilation tools in LIS use state-of-the art techniques such as 
the ensemble kalman filter to constrain the model behavior with observations. The coupled 
LIS- WRF system can be used to simultaneously employ both atmospheric data assimilation 
through WRF and land data assimilation through LIS. Such a state-of-the-art system could 
improve the predictive capabilities significantly in an operational environment. 

The coupling interface in the LIS- WRF system is currently implemented in an explicit man- 
ner, using the atmospheric and surface values for different time steps to compute the tur- 
bulent fluxes. The focus of this approach is to conserve energy. The fluxes fed back to the 
atmosphere is obtained by solving the surface energy balance. The main drawback is that 
the atmospheric feedback to the surface is only felt from one time step to the other. Other 
coupling strategies such as implicit, semi-implicit methods that focus on keeping the at- 
mospheric profiles synchronous to the surface conditions will be explored through future 
enhancements to the LIS- WRF system. Thus, the LIS- WRF system will enable studies to 
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investigate the nature of interaction and feedback between land and the atmosphere using 
various modeling tools and schema in LIS and WRF. 
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Table 1 

Comparison of Root Mean Square (RMS) and Bias errors for the initial soil moisture and soil 
temperature profiles at 12ZJune 12, 2002, used for the initialization of the precipitation forecasts 


1 

NARR 

RMS 

LIS-Noah 

LIS-CLM 

NARR 

Bias 

LIS-Noah 

LIS-CLM 

10 cm volumetric 







soil moisture (m3/m3) 

0.112 

0.058 

0.062 

-3.22E-2 

-1.42E-2 

4.17E-3 

30 cm volumetric 







soil moisture (m3/m3) 

0.129 

0.079 

0.087 

-7.47E-2 

-3.82E-2 

5.19E-2 

60 cm volumetric 







soil moisture (m3/m3) 

0.169 

0.090 

0.101 

-0.150 

-5.29E-2 

8.23E-2 

10 cm soil temperature (K) 

2.38 

1.37 

1.24 

1.32 

0.89 

-0.62 

30 cm soil temperature (K) 

2.41 

2.17 

.2.26 

1.29 

1.75 

-1.60 


Table 2 

Comparative skill scores from simulations using different land surface initial conditions compared 
with the STAGEIV data 


Metric 

NARR-based 

LIS-Noah 

LIS-CLM 

Root Mean Square Error (mm) 

5.97 

5.62 

5.38 

Bias Error (mm) 

8.85E-2 

2.70E-2 

-8.73E-2 

Probability of Detection (POD) 

0.21 

0.25 

0.35 

False Alarm Ratio (FAR) 

0.61 

0.56 

0.32 
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Fig. 2. Running modes of LIS in the LIS-WRF system: the uncoupled/analysis mode for initializa- 
tion and the coupled/forecast mode as the land modeling component 
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Fig. ,3. Sequence of component interactions in the coupled LIS-WRF system 
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Time (Hour:Minute) Time (Hour: Minute) 

Fig. 4. Domain averaged surface energy flux terms from the DRY simulations using: (a) Noah and 
(b) CLM land surface models 
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Fig. 5. Domain averaged surface energy flux terms from the WET simulations using: (a) Noah and 
(b) CLM land surface models 
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Fig. 6. Domain averaged PBL height from the DRY/WET simulations using: (a) Noah and (b) 
CLM land surface models 



Fig. 7. Configuration of the triple nested domain used in the coupled LIS-WRF simulations for the 
June 12, 2002 forecast 
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Fig. 8. Top. 10cm volumetric soil moisture initial conditions at 12Z, on June 12, 2002 generated 
by (a) WRF preprocessor (using NARR data), (b) long term spinup of Noah by LIS, and (c) long 
term spinup of CLM by LIS 






















Range of rain rates (mm/hr) 

Fig. 10. Comparison of the distribution of rain rates (mm/hr) from simulations using different land 
surface initial conditions compared with the STAGEIV data 
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Number of Processors 


Fig. 11. Computational scaling of the LIS-WRF system, compared with the native WRF system 
for a domain with 100x100 grid points at 1km resolution 





